Lightpainting live view

ABSTRACT

Methods and apparatus, including computer program products, for a light painting live view. A method includes, in a device comprising at least a processor, a memory, a display and a camera device having an on-screen viewfinder, accessing the camera, capturing individual frames of footage, each of the captured frames being displayed through the on-screen viewfinder in cumulative succession, rendering the captured frames on a graphical processing unit (GPU), sending the captured frames through a shader program, generating at least two images, a first image saved to the memory and a second image displayed on the display, and rendering the first image into the second image to generate a final image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.61/693,795, filed Aug. 28, 2012. The disclosure of the prior applicationis considered part of and is incorporated by reference in the disclosureof this application.

BACKGROUND OF THE INVENTION

The present invention generally relates to devices having a camerafeature, and more particularly to a light painting live view.

Like cameras, smartphones, such as the Apple iPhone®, Samsung Galaxy®,Blackberry Q10® and the like, and tablet computers running, for example,Google's Android® operating system (O/S) and Apple's iOS® O/S, includeamong their features, built-in cameras for taking photos. Applicationsexecuting in the smartphones and tablet computers enable control of thebuilt-in cameras, including light painting.

In general, light painting is a photographic technique, often performedat night or in a dark area, where a photographer can introduce differentlighting elements during a single long exposure photograph. lightpainting enables the capture of light trails, light graffiti tags, andso forth.

SUMMARY OF THE INVENTION

The following presents a simplified summary of the innovation in orderto provide a basic understanding of some aspects of the invention. Thissummary is not an extensive overview of the invention. It is intended toneither identify key or critical elements of the invention nor delineatethe scope of the invention. Its sole purpose is to present some conceptsof the invention in a simplified form as a prelude to the more detaileddescription that is presented later.

The present invention provides methods and apparatus, including computerprogram products, for a light painting live view.

In general, in one aspect, the invention features a method including, ina device including at least a processor, a memory, a display and acamera device having an on-screen viewfinder, accessing the camera,capturing individual frames of footage, each of the captured framesbeing displayed through the on-screen viewfinder in cumulativesuccession, rendering the captured frames on a graphical processing unit(GPU), sending the captured frames through a shader program, generatingat least two images, a first image saved to the memory and a secondimage displayed on the display, and rendering the first image into thesecond image to generate a final image

In another aspect, the invention features a method including, in adevice including at least a processor, a memory, a display and a cameradevice, executing a light painting live view process in conjunction withthe camera to provide a long exposure camera that displays a creation ofan exposure in real time.

In still another aspect, the invention features an apparatus including aprocessor, a memory, a display, and a camera device, the memoryincluding a light painting live view process, the light painting liveview process including accessing the camera, capturing individual framesof footage, each of the captured frames being displayed through theon-screen viewfinder in cumulative succession, rendering the capturedframes on a graphical processing unit (GPU), sending the captured framesthrough a shader program, generating at least two images, a first imagesaved to the memory and a second image displayed on the display, andrendering the first image into the second image to generate a finalimage.

These and other features and advantages will be apparent from a readingof the following detailed description and a review of the associateddrawings. It is to be understood that both the foregoing generaldescription and the following detailed description are explanatory onlyand are not restrictive of aspects as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be more fully understood by reference to the detaileddescription, in conjunction with the following figures, wherein:

FIG. 1 is a block diagram of an exemplary smartphone.

FIG. 2 is a flow diagram of an exemplary light painting live viewprocess.

DETAILED DESCRIPTION

The subject innovation is now described with reference to the drawings,wherein like reference numerals are used to refer to like elementsthroughout. In the following description, for purposes of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding of the present invention. It may be evident, however, thatthe present invention may be practiced without these specific details.In other instances, well-known structures and devices are shown in blockdiagram form in order to facilitate describing the present invention.

As used in this application, the terms “component,” “system,”“platform,” and the like can refer to a computer-related entity or anentity related to an operational machine with one or more specificfunctionalities. The entities disclosed herein can be either hardware, acombination of hardware and software, software, or software inexecution. For example, a component may be, but is not limited to being,a process running on a processor, a processor, an object, an executable,a thread of execution, a program, and/or a computer. By way ofillustration, both an application running on a server and the server canbe a component. One or more components may reside within a processand/or thread of execution and a component may be localized on onecomputer and/or distributed between two or more computers. Also, thesecomponents can execute from various computer readable media havingvarious data structures stored thereon. The components may communicatevia local and/or remote processes such as in accordance with a signalhaving one or more data packets (e.g., data from one componentinteracting with another component in a local system, distributedsystem, and/or across a network such as the Internet with other systemsvia the signal).

In addition, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom context, “X employs A or B” is intended to mean any of the naturalinclusive permutations. That is, if X employs A, X employs B, or Xemploys both A and B, then “X employs A or B” is satisfied under any ofthe foregoing instances. Moreover, articles “a” and “an” as used in thesubject specification and annexed drawings should generally be construedto mean “one or more” unless specified otherwise or clear from contextto be directed to a singular form.

As shown in FIG. 1, an exemplary device 10 includes at least a processor15, a memory 20, a display unit 25, a camera 30 and a graphicalprocessing unit (GPU) 35. Example devices 10 include DSLR cameras,smartphones, tablet computers, personal data assistants, digitaltelevisions, computers, laptops, devices with an integrated digitalcamera such as Nintendo® DS, wearable devices, devices with a digitalcamera, and so forth. The GPU 35 is an electronic circuit designed torapidly manipulate and alter memory 20 to accelerate a creation ofimages in a frame buffer intended for output to the display unit 25.

The memory 20 can include at least an operating system (O/S) 40, such asWindows®, Linux®, Google's Android®, Apple's iOS®, or a proprietary OS,and a light painting live view process 100.

Light painting is a photographic technique in which exposures are madeby moving a hand-held light source or by moving the camera. The termlight painting also encompasses images lit from outside the frame withhand-held light sources. By moving the light source, the light can beused to selectively illuminate parts of the subject or to “paint” apicture by shining it directly into the camera lens. Light paintingrequires a slow shutter speed, usually a second or more. Light paintingcan take on the characteristics of a quick pencil sketch.

Light painting by moving the camera, also called camera painting, is theantithesis of traditional photography. At night, or in a dark room, thecamera can be taken off the tripod and used like a paintbrush. Anexample is using the night sky as the canvas, the camera as the brushand cityscapes (amongst other light sources) as the palette. Puttingenergy into moving the camera by stroking lights, making patterns andlaying down backgrounds can create abstract artistic images.

Light painting can be done interactively using a webcam. The paintedimage can already be seen while drawing by using a monitor or projector.

Another technique used in the creation of light art is the projection ofimages on to irregular surfaces (faces, bodies, buildings, and soforth), in effect “painting” them with light. A photograph or otherfixed portrayal of the resulting image is then made.

The light painting live view process 100 executes in conjunction withthe camera 30 to provide a long exposure camera that displays thecreation of the exposure in real time.

The device 10 can support a variety of applications, such as a telephoneapplication, a video conferencing application, an e-mail application, aninstant messaging application, a blogging application, a web browsingapplication, a digital music player application, and/or a digital videoplayer application.

The light painting live view process 100 is a light paintingapplication. In light painting, a user can use a light source to drawshapes and patterns in front of a camera set to a long exposure. Thelight painting live view process 100 enables the user behind the camera30 within the device 10 (or tablet computer) to watch the shapes orpatterns that are being created, as they are being created. In priorapproaches, the user must wait until the end of the exposure to see whathas been made or created.

As shown om FIG. 2, the light painting live view process 100 accesses(105) the camera, which captures individual frames of footage, each ofthe captured frames displayed on a viewfinder in cumulative succession.

While frames are being captured by the camera, the light painting liveview process 100 renders (110) the captured frames on a graphicalprocessing unit (GPU), which is a user-facing camera “viewfinder”feature of the light painting live view process 100.

For every frame that is being captured to create an image, the lightpainting live view process 100 also sends (115) them through a shaderprogram (also referred to as a vertex and fragment program) intographical processing unit (GPU). In general, a shader is a computerprogram that is used to do shading, produce special effects and/or dopost-processing. Shaders calculate rendering effects on graphicshardware with a high degree of flexibility. Most shaders are coded for agraphics processing unit (GPU), though this is not a strict requirement.The position, hue, saturation, brightness, and contrast of all pixels,vertices, or textures used to construct a final image can be altered onthe fly, using algorithms defined in the shader, and can be modified byexternal variables or textures introduced by the program calling theshader.

Sending (115) the captured frames through the shader creates two images,one image saved (120) to the device's memory and the other imagedisplayed (125) by light painting live view process 100 for the user tosee as if they were watching a video. The light painting live viewprocess 100 uses frames from the camera as the input of the shaderprogram and a progress frame as the output of the shader program.Through additive blending, one image is rendered (130) into the other bythe light painting live view process 100, i.e., the image that is beingdrawn progressively is rendered to the display.

Once the user signals the light painting live view process to stop, thelight painting live view process 100 converts (135) the image that isrendered into the memory to a Joint Photographic Experts Group (JPEG)file and projects (140) the JPEG file as a final image on the display.

As described above, a user initiates the light painting live viewprocess 100, which generates a home screen graphical user interface(GUI). The GUI includes a main navigation bar that includes a pictorialrendering of a small camera. When the small camera is tapped, the lightpainting live view process 100 opens up to the camera built into thedevice's memory. The camera screen appears as though it's a videoscreen, ready for capture. The navigation bar shows a button to tap tobegin image capture.

A video capture session is initiated and anything that passes in frontof the camera will leave a trail, similar to a long exposure on asingle-lens reflex/digital single-lens reflex (SLR/DSLR) camera. Thedifference is that the user sees the trail as it is created, in realtime, like a mixture of a stop motion video and an Etch-A-Sketch®.

This is viewed facing through the viewfinder on of the light paintinglive view process 100, which is a screen that accesses the forwardfacing camera on the device. Anything viewed by that camera is seenthrough the light painting live view process 100 viewfinder.

Exposures can be set for one second, or they can run as long as the userhas memory in their device to store the image/video data. The exposurecan also be stopped by tapping the same button used to start theexposure.

The user can move their camera around to capture trails, or they canmake their own trails with a light of their own.

For every frame that is being captured to create the image, the capturedframe is sent through a shader program into the GPU.

A GL_MAX blend operation, which specifies how source and destinationcolors are combined, is responsible for producing the light painting,but to control the output a fragment shader program is used. Thefragment shader is run on each pixel of an image, producing for eachinput pixel a corresponding output pixel. The fragment shader supportsan “Ambient Light Amount” feature of the capture settings. By taking abrightness parameter between 0 and 1, the fragment shader enablesthrottling the affect of light input on the painting.

The following is one example of fragment shader source code:

precision mediump float;

varying vec2 v_uv;

niform sampler2D u_diffuseTexture;

uniform float u_brightness;

void main(void)

{

-   -   //sample color    -   vec4 color=texture2D(u_diffuseTexture, v_uv);    -   //calculate luminance intensity    -   float lumIntensity=color.x*0.299+color.y*0.587+color.z*0.114;    -   //clamp and exaggerate luminance intensity    -   lumIntensity=min(1.0, lumIntensity);    -   lumIntensity=lumIntensity*lumIntensity;    -   lumIntensity=max(u_brightness, lumIntensity);    -   //output final color    -   gl_FragColor=color*lumIntensity;

}

The light painting live view process 100 then generates images instages:

Image Stages/Names Stage

1. Raw Image—this is the image data coming from the device's videocamera, frame-by-frame, stored in a buffer managed by the operatingsystem.

2. Input Image—this is the image used as an input to the fragment shaderprogram, stored in an OpenGL texture. A texture is an OpenGL Object thatcontains one or more images that all have the same image format. Atexture can be used in two ways. It can be the source of a textureaccess from a shader, or it can be used as a render target. The rawimage is copied into the input image.

3. Intermediate Output Image—this is the output of the fragment shaderprogram, stored in an OpenGL texture. The input image is rendered intothe intermediate output image, using a custom OpenGL frame buffer backedby an OpenGL texture. In general, frame buffer objects are a mechanismfor rendering to images other than the default OpenGL Default framebuffer. They are OpenGL Objects that allow you to render directly totextures, as well as blitting from one frame buffer. to another.

4. Preview Image—this is the output of the fragment shader program,shown on the device's display. The input image is rendered to thescreen, using the default OpenGL frame buffer backed by the device'sdisplay.

5. Output Image—this is the output of copying and compressing the datafrom the intermediate output image to a JPEG representation. The outputimage may be saved to the device's display's camera roll, shared viaemail, Facebook® or Twitter®, or uploaded to a server.

Through additive blending, one image is rendered into the other in theorder laid out above. The image that is being drawn progressively isrendered to the display.

Blending Modes Stage

To produce a light painting, the pixels of the intermediate output imageare blended with the pixels of the input image. The output of thatblending process is then used to replace the previous value of eachpixel of the intermediate output image.

The OpenGL blend mode “GL_MAX” is used to blend the pixels. The maximumof the two pixel values is the output of the operation.

The following describes the effect of the GL_MAX blend mode on pixelvalues (taken from the OpenGL documentation at

-   http://www.opengl.org/sdk/docs/man/xhtml/glBlendEquation.xml):

Mode

RGB Components

Alpha Component

GL_MAX

Rr=max

s R d

Gr=max

s G d

Br=max

s B d

Ar=max

s A d

When all done, the output image is displayed:

5. Output Image—this is the output of copying and compressing the datafrom the intermediate output image to a JPEG representation. The outputimage may be saved to the device's display's camera roll, shared viaemail, Facebook® or Twitter®, or uploaded to the server.

While the above describes a particular order of operations performed bycertain embodiments of the invention, it should be understood that suchorder is exemplary, as alternative embodiments may perform theoperations in a different order, combine certain operations, overlapcertain operations, or the like. References in the specification to agiven embodiment indicate that the embodiment described may include aparticular feature, structure, or characteristic, but every embodimentmay not necessarily include the particular feature, structure, orcharacteristic.

While given components of the system have been described separately, oneof ordinary skill will appreciate that some of the functions may becombined or shared in given instructions, program sequences, codeportions, and the like.

The foregoing description does not represent an exhaustive list of allpossible implementations consistent with this disclosure or of allpossible variations of the implementations described. A number ofimplementations have been described. Nevertheless, it will be understoodthat various modifications may be made without departing from the spiritand scope of the systems, devices, methods and techniques describedhere. Accordingly, other implementations are within the scope of thefollowing claims.

What is claimed is:
 1. A method comprising: in a device comprising atleast a processor, a memory, an integrated display and an integratedcamera device having an on-screen viewfinder, accessing the integratedcamera; continuously capturing individual frames of footage and thendisplaying a progress of the individual frames that create a final imageas they are captured, in real time, with each frame displaying as it iscaptured in cumulative succession on the on-screen viewfinder as thecapture is being made, such that any light or object which passes infront of the camera will leave a trail on the on-screen viewfinder thatillustrates a path that the light or object traveled during an exposureperiod until a user stops capturing or the device ceases to function;rendering the captured frames on a graphical processing unit (GPU);sending the captured frames through a shader program; generating atleast two images, a first image saved to the memory and a second imagedisplayed on the on-screen viewfinder; and rendering the first imageinto the second image to generate a final image.
 2. The method of claim1 wherein the shader program receives input from the camera and outputsa progress frame.
 3. The method of claim 2 wherein generating the atleast two images comprises: an image/name stage; and blending modesstage.
 4. The method of claim 3 wherein the image/name stage comprises:storing image data coming from the camera in a buffer in the memory;using the stored image as an input to the shader program; and outputtingan intermediate image from the shader program to the display, theintermediate image blended with the input images.
 5. The method of claim4 wherein the blending modes stage comprises: blending pixels of theintermediate output image with pixels of the input image; and replacingprevious values of pixels with pixels of the intermediate output image.6. The method of claim 5 wherein the input is a OpenGL texture.
 7. Themethod of claim 6 further comprising: compressing the final image; andconverting the compressed final image to a Joint Photographic ExpertsGroup (JPEG) file.
 8. The method of claim 7 further comprisingprojecting the JPEG file on the display.
 9. The method of claim 8wherein the device is selected from the group consisting of a DSLRcamera, a smartphone, a tablet computer, a personal data assistants, adigital televisions, a computers, a laptops, a device with an integrateddigital camera, a wearable device, and a device with a digital camera, adigital camera, and a personal data assistant.
 10. The method of claim 9wherein the device is a smartphone or tablet computer.
 11. The method ofclaim 10, wherein the blending modes stage further comprises: selectinga maximum value between each of a plurality of pixel values of eachpixel of the input image and each corresponding pixel value of eachcorresponding pixel of the intermediate output image; and outputtingeach maximum pixel value as the pixel value for each corresponding pixelof the resultant blended image.
 12. The method of claim 11, wherein theshader program further comprises: measuring an ambient light amount forthe input image; setting a brightness parameter between 0 and 1; andthrottling the effect of light input on a light painting.
 13. Anapparatus comprising: a processor; a memory; a display; and a cameradevice having an on-screen viewfinder; the memory comprising a lightpainting live view process, the light painting live view processcomprising the steps of the method of claim 1.